{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# This jupyter notebook documents the efforts to read the mat files from CALCE website. \n",
    "\n",
    "The data is located [here](https://web.calce.umd.edu/batteries/data.htm#pls) from experiments on various cells called 'PL Samples' - PL 3, 10 ,4, 5 , 9, 25 etc. After unzipping the files, give the path to the file stored. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Replace this with the path to the downloaded mat files in your system\n",
    "matdata_dir = '/home/ubuntu_cp/uwdirect/BattDeg/data/raw_mat_PL0310/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the necessary packages\n",
    "import scipy.io as spio\n",
    "from os.path import isfile, join\n",
    "import numpy as np\n",
    "import h5py \n",
    "import tables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. `scipy.io` seems to be one of the most popular methods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the file into a python object\n",
    "mat = spio.loadmat(join(matdata_dir, 'PL03.mat'), squeeze_me=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'__header__': b'MATLAB 5.0 MAT-file, Platform: PCWIN64, Created on: Mon Feb 08 20:42:18 2016',\n",
       " '__version__': '1.0',\n",
       " '__globals__': [],\n",
       " 'PL03': array([['Operation', 'Start Date', 'Data'],\n",
       "        ['Single Cycle', 'December 09, 2014',\n",
       "         MatlabOpaque([(b'', b'MCOS', b'table', array([3707764736,          2,          1,          1,          1,\n",
       "                 1], dtype=uint32))],\n",
       "              dtype=[('s0', 'O'), ('s1', 'O'), ('s2', 'O'), ('arr', 'O')])],\n",
       "        ['75 Partial Cycles', 'December 10, 2014',\n",
       "         MatlabOpaque([(b'', b'MCOS', b'table', array([3707764736,          2,          1,          1,          2,\n",
       "                 1], dtype=uint32))],\n",
       "              dtype=[('s0', 'O'), ('s1', 'O'), ('s2', 'O'), ('arr', 'O')])],\n",
       "        ['75 Partial Cycles', 'December 30, 2014',\n",
       "         MatlabOpaque([(b'', b'MCOS', b'table', array([3707764736,          2,          1,          1,          3,\n",
       "                 1], dtype=uint32))],\n",
       "              dtype=[('s0', 'O'), ('s1', 'O'), ('s2', 'O'), ('arr', 'O')])],\n",
       "        ['50 Partial Cycles', 'January 14, 2015',\n",
       "         MatlabOpaque([(b'', b'MCOS', b'table', array([3707764736,          2,          1,          1,          4,\n",
       "                 1], dtype=uint32))],\n",
       "              dtype=[('s0', 'O'), ('s1', 'O'), ('s2', 'O'), ('arr', 'O')])],\n",
       "        ['50 Partial Cycles', 'February 05, 2015',\n",
       "         MatlabOpaque([(b'', b'MCOS', b'table', array([3707764736,          2,          1,          1,          5,\n",
       "                 1], dtype=uint32))],\n",
       "              dtype=[('s0', 'O'), ('s1', 'O'), ('s2', 'O'), ('arr', 'O')])],\n",
       "        ['50 Partial Cycles', 'March 12, 2015',\n",
       "         MatlabOpaque([(b'', b'MCOS', b'table', array([3707764736,          2,          1,          1,          6,\n",
       "                 1], dtype=uint32))],\n",
       "              dtype=[('s0', 'O'), ('s1', 'O'), ('s2', 'O'), ('arr', 'O')])],\n",
       "        ['50 Partial Cycles', 'March 26, 2015',\n",
       "         MatlabOpaque([(b'', b'MCOS', b'table', array([3707764736,          2,          1,          1,          7,\n",
       "                 1], dtype=uint32))],\n",
       "              dtype=[('s0', 'O'), ('s1', 'O'), ('s2', 'O'), ('arr', 'O')])],\n",
       "        ['50 Partial Cycles', 'April 14, 2015',\n",
       "         MatlabOpaque([(b'', b'MCOS', b'table', array([3707764736,          2,          1,          1,          8,\n",
       "                 1], dtype=uint32))],\n",
       "              dtype=[('s0', 'O'), ('s1', 'O'), ('s2', 'O'), ('arr', 'O')])],\n",
       "        ['50 Partial Cycles', 'May 06, 2015',\n",
       "         MatlabOpaque([(b'', b'MCOS', b'table', array([3707764736,          2,          1,          1,          9,\n",
       "                 1], dtype=uint32))],\n",
       "              dtype=[('s0', 'O'), ('s1', 'O'), ('s2', 'O'), ('arr', 'O')])],\n",
       "        ['50 Partial Cycles', 'June 19, 2015',\n",
       "         MatlabOpaque([(b'', b'MCOS', b'table', array([3707764736,          2,          1,          1,         10,\n",
       "                 1], dtype=uint32))],\n",
       "              dtype=[('s0', 'O'), ('s1', 'O'), ('s2', 'O'), ('arr', 'O')])],\n",
       "        ['50 Partial Cycles', 'July 01, 2015',\n",
       "         MatlabOpaque([(b'', b'MCOS', b'table', array([3707764736,          2,          1,          1,         11,\n",
       "                 1], dtype=uint32))],\n",
       "              dtype=[('s0', 'O'), ('s1', 'O'), ('s2', 'O'), ('arr', 'O')])],\n",
       "        ['50 Partial Cycles', 'July 13, 2015',\n",
       "         MatlabOpaque([(b'', b'MCOS', b'table', array([3707764736,          2,          1,          1,         12,\n",
       "                 1], dtype=uint32))],\n",
       "              dtype=[('s0', 'O'), ('s1', 'O'), ('s2', 'O'), ('arr', 'O')])],\n",
       "        ['50 Partial Cycles', 'July 21, 2015',\n",
       "         MatlabOpaque([(b'', b'MCOS', b'table', array([3707764736,          2,          1,          1,         13,\n",
       "                 1], dtype=uint32))],\n",
       "              dtype=[('s0', 'O'), ('s1', 'O'), ('s2', 'O'), ('arr', 'O')])],\n",
       "        ['50 Partial Cycles', 'July 29, 2015',\n",
       "         MatlabOpaque([(b'', b'MCOS', b'table', array([3707764736,          2,          1,          1,         14,\n",
       "                 1], dtype=uint32))],\n",
       "              dtype=[('s0', 'O'), ('s1', 'O'), ('s2', 'O'), ('arr', 'O')])],\n",
       "        ['50 Partial Cycles (Data Missing)', 'August 15, 2015',\n",
       "         array([], dtype=float64)],\n",
       "        ['50 Partial Cycles (Data Missing)', 'August 30, 2015',\n",
       "         array([], dtype=float64)],\n",
       "        ['Single Cycle', 'September 14, 2015',\n",
       "         MatlabOpaque([(b'', b'MCOS', b'table', array([3707764736,          2,          1,          1,         15,\n",
       "                 1], dtype=uint32))],\n",
       "              dtype=[('s0', 'O'), ('s1', 'O'), ('s2', 'O'), ('arr', 'O')])],\n",
       "        ['50 Partial Cycles', 'September 19, 2015',\n",
       "         MatlabOpaque([(b'', b'MCOS', b'table', array([3707764736,          2,          1,          1,         16,\n",
       "                 1], dtype=uint32))],\n",
       "              dtype=[('s0', 'O'), ('s1', 'O'), ('s2', 'O'), ('arr', 'O')])],\n",
       "        ['50 Partial Cycles', 'October 09, 2015',\n",
       "         MatlabOpaque([(b'', b'MCOS', b'table', array([3707764736,          2,          1,          1,         17,\n",
       "                 1], dtype=uint32))],\n",
       "              dtype=[('s0', 'O'), ('s1', 'O'), ('s2', 'O'), ('arr', 'O')])],\n",
       "        ['50 Partial Cycles', 'October 27, 2015',\n",
       "         MatlabOpaque([(b'', b'MCOS', b'table', array([3707764736,          2,          1,          1,         18,\n",
       "                 1], dtype=uint32))],\n",
       "              dtype=[('s0', 'O'), ('s1', 'O'), ('s2', 'O'), ('arr', 'O')])],\n",
       "        ['50 Partial Cycles', 'November 25, 2015',\n",
       "         MatlabOpaque([(b'', b'MCOS', b'table', array([3707764736,          2,          1,          1,         19,\n",
       "                 1], dtype=uint32))],\n",
       "              dtype=[('s0', 'O'), ('s1', 'O'), ('s2', 'O'), ('arr', 'O')])],\n",
       "        ['50 Partial Cycles', 'December 04, 2015',\n",
       "         MatlabOpaque([(b'', b'MCOS', b'table', array([3707764736,          2,          1,          1,         20,\n",
       "                 1], dtype=uint32))],\n",
       "              dtype=[('s0', 'O'), ('s1', 'O'), ('s2', 'O'), ('arr', 'O')])],\n",
       "        ['50 Partial Cycles', 'December 14, 2015',\n",
       "         MatlabOpaque([(b'', b'MCOS', b'table', array([3707764736,          2,          1,          1,         21,\n",
       "                 1], dtype=uint32))],\n",
       "              dtype=[('s0', 'O'), ('s1', 'O'), ('s2', 'O'), ('arr', 'O')])],\n",
       "        ['100 Partial Cycles', 'December 23, 2015',\n",
       "         MatlabOpaque([(b'', b'MCOS', b'table', array([3707764736,          2,          1,          1,         22,\n",
       "                 1], dtype=uint32))],\n",
       "              dtype=[('s0', 'O'), ('s1', 'O'), ('s2', 'O'), ('arr', 'O')])]],\n",
       "       dtype=object),\n",
       " '__function_workspace__': array([[ 0,  1, 73, ...,  0,  0,  0]], dtype=uint8)}"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Display the contents as read\n",
    "mat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check the type of the object\n",
    "type(mat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### For the dictionary object, read the keys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['__header__', '__version__', '__globals__', 'PL03', '__function_workspace__'])"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mat.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the data in key 'PL03'\n",
    "PL03 = mat['PL03']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "numpy.ndarray"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the type of this object\n",
    "type(PL03)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Inspect the numpy array, starting form the second element, where all the cycling data is stored\n",
    "pl03_1 = PL03[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['Single Cycle', 'December 09, 2014',\n",
       "       MatlabOpaque([(b'', b'MCOS', b'table', array([3707764736,          2,          1,          1,          1,\n",
       "                1], dtype=uint32))],\n",
       "             dtype=[('s0', 'O'), ('s1', 'O'), ('s2', 'O'), ('arr', 'O')])],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pl03_1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### THis is where we hit the wall, as apparently scipy is trying to hint us with the name of the object - `MatlabOpaque` and not a lot of details in the documentation. \n",
    "\n",
    "> \"Subclass to signal this is a matlab opaque matrix\" [1](https://www.pydoc.io/pypi/scipy-1.0.0rc2/autoapi/io/matlab/mio5_params/index.html#io.matlab.mio5_params.MatlabOpaque.__new__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Single Cycle',\n",
       " 'December 09, 2014',\n",
       " MatlabOpaque([(b'', b'MCOS', b'table', array([3707764736,          2,          1,          1,          1,\n",
       "                 1], dtype=uint32))],\n",
       "              dtype=[('s0', 'O'), ('s1', 'O'), ('s2', 'O'), ('arr', 'O')])]"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Trying various functions that are available with this object as listed here https://kite.com/python/docs/scipy.io.matlab.mio5_params.MatlabOpaque\n",
    "pl03_1.tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### In other words, a deadend, as confirmed here: https://stackoverflow.com/questions/15512560/access-mat-file-containing-matlab-classes-in-python. Some success has been achieved in reading the matlab files in Julia by the author, but that seems like a big aside for this project. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__________________________________________________"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Trying other packages in python all result in error \n",
    "### 2.1 `mat4py`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "ename": "ParseError",
     "evalue": "Got type 1, expected 5 (miINT32)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mParseError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-78-55472fb8d3f0>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mmat4py\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mloadmat\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mdata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mloadmat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmatdata_dir\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'PL03.mat'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/anaconda3/envs/keras2/lib/python3.6/site-packages/mat4py/loadmat.py\u001b[0m in \u001b[0;36mloadmat\u001b[0;34m(filename, meta)\u001b[0m\n\u001b[1;32m    461\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    462\u001b[0m         \u001b[0;31m# read the matrix\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 463\u001b[0;31m         \u001b[0mmdict\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mread_var_array\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfd_var\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mendian\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhdr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    464\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mmeta\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mhdr\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'is_global'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    465\u001b[0m             \u001b[0mmdict\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'__globals__'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/keras2/lib/python3.6/site-packages/mat4py/loadmat.py\u001b[0m in \u001b[0;36mread_var_array\u001b[0;34m(fd, endian, header)\u001b[0m\n\u001b[1;32m    366\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mread_char_array\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfd\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mendian\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mheader\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    367\u001b[0m     \u001b[0;32melif\u001b[0m \u001b[0mmc\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m'mxCELL_CLASS'\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 368\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mread_cell_array\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfd\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mendian\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mheader\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    369\u001b[0m     \u001b[0;32melif\u001b[0m \u001b[0mmc\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m'mxSTRUCT_CLASS'\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    370\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mread_struct_array\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfd\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mendian\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mheader\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/keras2/lib/python3.6/site-packages/mat4py/loadmat.py\u001b[0m in \u001b[0;36mread_cell_array\u001b[0;34m(fd, endian, header)\u001b[0m\n\u001b[1;32m    292\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mcol\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mheader\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'dims'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    293\u001b[0m             \u001b[0;31m# read the matrix header and array\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 294\u001b[0;31m             \u001b[0mvheader\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnext_pos\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfd_var\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mread_var_header\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfd\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mendian\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    295\u001b[0m             \u001b[0mvarray\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mread_var_array\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfd_var\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mendian\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvheader\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    296\u001b[0m             \u001b[0marray\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mrow\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvarray\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/keras2/lib/python3.6/site-packages/mat4py/loadmat.py\u001b[0m in \u001b[0;36mread_var_header\u001b[0;34m(fd, endian)\u001b[0m\n\u001b[1;32m    250\u001b[0m                          'got {}'.format(etypes['miMATRIX']['n'], mtpn))\n\u001b[1;32m    251\u001b[0m     \u001b[0;31m# read the header\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 252\u001b[0;31m     \u001b[0mheader\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mread_header\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfd\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mendian\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    253\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mheader\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnext_pos\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfd\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    254\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/keras2/lib/python3.6/site-packages/mat4py/loadmat.py\u001b[0m in \u001b[0;36mread_header\u001b[0;34m(fd, endian)\u001b[0m\n\u001b[1;32m    215\u001b[0m         \u001b[0;34m'nzmax'\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mnzmax\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    216\u001b[0m     }\n\u001b[0;32m--> 217\u001b[0;31m     \u001b[0mheader\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'dims'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mread_elements\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfd\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mendian\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m'miINT32'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    218\u001b[0m     \u001b[0mheader\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'n_dims'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mheader\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'dims'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    219\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mheader\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'n_dims'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0;36m2\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/keras2/lib/python3.6/site-packages/mat4py/loadmat.py\u001b[0m in \u001b[0;36mread_elements\u001b[0;34m(fd, endian, mtps, is_name)\u001b[0m\n\u001b[1;32m    178\u001b[0m         raise ParseError('Got type {}, expected {}'.format(\n\u001b[1;32m    179\u001b[0m             mtpn, ' / '.join('{} ({})'.format(\n\u001b[0;32m--> 180\u001b[0;31m                 etypes[mtp]['n'], mtp) for mtp in mtps)))\n\u001b[0m\u001b[1;32m    181\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mdata\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    182\u001b[0m         \u001b[0;31m# full format, read data\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mParseError\u001b[0m: Got type 1, expected 5 (miINT32)"
     ]
    }
   ],
   "source": [
    "from mat4py import loadmat\n",
    "data = loadmat(join(matdata_dir, 'PL03.mat'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 `h5py`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "ename": "OSError",
     "evalue": "Unable to open file (file signature not found)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mOSError\u001b[0m                                   Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-93-25479f2c450c>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mh5py\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mFile\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmatdata_dir\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'PL03.mat'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'r'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/anaconda3/envs/keras2/lib/python3.6/site-packages/h5py/_hl/files.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, name, mode, driver, libver, userblock_size, swmr, rdcc_nslots, rdcc_nbytes, rdcc_w0, track_order, **kwds)\u001b[0m\n\u001b[1;32m    392\u001b[0m                 fid = make_fid(name, mode, userblock_size,\n\u001b[1;32m    393\u001b[0m                                \u001b[0mfapl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfcpl\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mmake_fcpl\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtrack_order\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtrack_order\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 394\u001b[0;31m                                swmr=swmr)\n\u001b[0m\u001b[1;32m    395\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    396\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mswmr_support\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/keras2/lib/python3.6/site-packages/h5py/_hl/files.py\u001b[0m in \u001b[0;36mmake_fid\u001b[0;34m(name, mode, userblock_size, fapl, fcpl, swmr)\u001b[0m\n\u001b[1;32m    168\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mswmr\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mswmr_support\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    169\u001b[0m             \u001b[0mflags\u001b[0m \u001b[0;34m|=\u001b[0m \u001b[0mh5f\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mACC_SWMR_READ\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 170\u001b[0;31m         \u001b[0mfid\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mh5f\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mflags\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfapl\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mfapl\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    171\u001b[0m     \u001b[0;32melif\u001b[0m \u001b[0mmode\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m'r+'\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    172\u001b[0m         \u001b[0mfid\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mh5f\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mh5f\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mACC_RDWR\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfapl\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mfapl\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32mh5py/_objects.pyx\u001b[0m in \u001b[0;36mh5py._objects.with_phil.wrapper\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32mh5py/_objects.pyx\u001b[0m in \u001b[0;36mh5py._objects.with_phil.wrapper\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32mh5py/h5f.pyx\u001b[0m in \u001b[0;36mh5py.h5f.open\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;31mOSError\u001b[0m: Unable to open file (file signature not found)"
     ]
    }
   ],
   "source": [
    " h5py.File(join(matdata_dir, 'PL03.mat'),'r')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " This error stems from the fact that `h5py` can read HDF5 format files, however our mat file since it is MATLAB 5.0 file, as is evedent in the first few lines of this file, cannot be therefore read by `h5py` as confirmed [here](https://stackoverflow.com/questions/38089950/error-opening-file-in-h5py-file-signature-not-found)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 `pytables`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "ename": "HDF5ExtError",
     "evalue": "HDF5 error back trace\n\n  File \"H5F.c\", line 511, in H5Fopen\n    unable to open file\n  File \"H5Fint.c\", line 1604, in H5F_open\n    unable to read superblock\n  File \"H5Fsuper.c\", line 413, in H5F__super_read\n    file signature not found\n\nEnd of HDF5 error back trace\n\nUnable to open/create file '/home/ubuntu_cp/uwdirect/BattDeg/data/raw_mat_PL0310/PL03.mat'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mHDF5ExtError\u001b[0m                              Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-94-98265af88064>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# Trying another package\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mtables\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mopen_file\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmatdata_dir\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'PL03.mat'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/anaconda3/envs/keras2/lib/python3.6/site-packages/tables/file.py\u001b[0m in \u001b[0;36mopen_file\u001b[0;34m(filename, mode, title, root_uep, filters, **kwargs)\u001b[0m\n\u001b[1;32m    318\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    319\u001b[0m     \u001b[0;31m# Finally, create the File instance, and return it\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 320\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0mFile\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfilename\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtitle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mroot_uep\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfilters\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    321\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    322\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/keras2/lib/python3.6/site-packages/tables/file.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, filename, mode, title, root_uep, filters, **kwargs)\u001b[0m\n\u001b[1;32m    782\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    783\u001b[0m         \u001b[0;31m# Now, it is time to initialize the File extension\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 784\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_g_new\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfilename\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mparams\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    785\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    786\u001b[0m         \u001b[0;31m# Check filters and set PyTables format version for new files.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32mtables/hdf5extension.pyx\u001b[0m in \u001b[0;36mtables.hdf5extension.File._g_new\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;31mHDF5ExtError\u001b[0m: HDF5 error back trace\n\n  File \"H5F.c\", line 511, in H5Fopen\n    unable to open file\n  File \"H5Fint.c\", line 1604, in H5F_open\n    unable to read superblock\n  File \"H5Fsuper.c\", line 413, in H5F__super_read\n    file signature not found\n\nEnd of HDF5 error back trace\n\nUnable to open/create file '/home/ubuntu_cp/uwdirect/BattDeg/data/raw_mat_PL0310/PL03.mat'"
     ]
    }
   ],
   "source": [
    "# Trying another package\n",
    "tables.open_file(join(matdata_dir, 'PL03.mat'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again since this is not a HDF5 format file. It would have ben good if matlab provided an online file converter for their old mat files, but I did not find any today. There was another suggestion of reading these files as binary files [here](https://stackoverflow.com/a/26295900/1328232), however that seems like a gross overkill for this project too, though quite interesting. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Converting the files on matlab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `The .mat files were successfully coverted on matlab using the following code`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "load mat_file_name  \n",
    "writetable(old_table_name,'new_csv_file_name.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `Following code is an example and shows how the tables in the .mat files can be converted to csv files`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example: Converting a table names PL03{2,3} in the file PL03 to PL03(2).csv\n",
    "load PL03  \n",
    "writetable(PL03{2,3},'PL03(2).csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
